When an observer measures the angle to a star (including the sun), the sextant is held at some height above sea level. Since the Earth curves away at the sea's horizon, the sea horizon is necessarily down from the observers eye. Figure 1 is an exagerated view of this concept. When the DIP is obtained from a Nautical almanac it includes a correction for refraction, which is not shown here. That correction is usually very small as long as the celestial object is more than about $35^\circ$ high - on the order of 0.3 minutes.
Figure 1: In this exaggerated diagram, the angle measured by the sextant is from the star to the observed horizon. It includes the True Angle to the star as well as the DIP angle. We must subtract the DIP angle from the angle measured by the sextant. It is one of possibly several corrections that need to be made before going to an almanac. This article will explain how to compute the DIP angle.
Given Earth, with radius R=6371 km center, C at (0,0). For now, we can work
in two dimensions. Suppose that an observer is on top of a mountain that is 2 km high.
Figure 2: Earth's curve is still highly exaggerated in order to show the triangle legs. At the correct scale, length h would be 1000 times longer! Distance 'd' can be calculated since this is a right triangle. R is known and here, 'h' is the observers height plus Earth's radius. Angle $\alpha$ is calculated and used to establish coordinates for P.
At point P, which marks the horizon for our observer, there is a right triangle. We know Earths radius,
$R=6371,$ and we know that the observer is 2 km further up, so $h=6373$ km. By Pathagoras, we can calculate "d".
$$d=\sqrt{h^2-R^2}.$$
Having found "d", we can use trigonometry to say that $\alpha = tan^{-1}\frac{R}{d}$
There are many ways to establish coordinates for point P, but one of the easiest is to establish new basis vectors.
$$\mathbf{u}=\frac{A-C}{|A-C|}$$
$$\mathbf{e_1 = u} \qquad \mathbf{e_2=\left(y(u), -x(u)\right)}$$
Build the vector with angle $\alpha$.
$$\mathbf{v} = cos(\alpha)\cdot\mathbf{e_1}+sin(\alpha)\cdot\mathbf{e_2}$$
Then, $P=A-d\cdot\mathbf{v}.$
Now suppose we use a sextant to measure the angle to star B. Using our horizon, not the "Level Horizon Line", but rather the observer horizon we measure an angle of $58^\circ 29.16'$
Figure 3: An observer will not have access to the "Level Horizon Line". The angle between the level horizon line and the observed horizon is known as the dip. When the dip is obtained from a Nautical almanac it includes a correction for refraction, which is not shown here, but which is normally quite small. The dip, plus the refraction has to be subtracted from $h_s$ which is the measured angle $\beta$ in this figure.
We want to calculate the DIP as a function of the observers height above sea level. In the figure, sea level would be the "Level Horizon Line". Since we have already found 'P', which is the Earth horizon edge that can be seen from our elevation, and since we know the observer's height, we can imagine a Level line at the observer's height and use that to compute the DIP angle.
Let vector $\ell_1 = \binom{-1}{0},$ which is the same as
$$\ell_1 = \frac{\binom{P_x-A_x}{A_y-A_y}}{\left|\binom{P_x}{A_y}-\binom{A_x}{A_y} \right|}$$
Let vector $\ell_2 = (P-A) = \frac{\binom{P_x}{P_y}-\binom{A_x}{A_y}}{\left| \binom{P_x}{P_y}-\binom{A_x}{A_y}\right|}$
Now we can use the law of cosines for vectors.
$$\ell_1\cdot\ell_2 = |\ell_1|\cdot|\ell_2|\cdot\cos\theta$$
to get the cosine of $\theta$ which here is the DIP angle. So,
$$\angle DIP = \cos^{-1}\frac{\ell_1\cdot\ell_2}{|\ell_1|\cdot|\ell_2|}$$
Throughout all of this, it is good to remember that our height above sea level is
$$A_y - \text{ Earth's Radius}.$$
